Add Custom Metadata overlay for SDK groups and PropertyDefinition narrowing by chinmaya-singal-glean · Pull Request #101 · gleanwork/open-api

chinmaya-singal-glean · 2026-05-07T16:29:31Z

Summary

Groups all 5 Custom Metadata endpoints under indexing.customMetadata in generated SDKs and applies clean method names (upsert, delete, getSchema, upsertSchema, deleteSchema).
Introduces a CustomMetadataPropertyDefinition schema that exposes only name, propertyType, and skipIndexing, and re-points CustomMetadataSchema.metadataKeys.items to it. The shared PropertyDefinition schema (used by the Datasource / Custom Properties API) carries fields like displayLabel, displayLabelPlural, uiOptions, hideUiFacet, uiFacetOrder, group that are not relevant to Custom Metadata. We can't strip them from PropertyDefinition itself without affecting the Indexing API surface, so we narrow via a separate schema.
Registers the overlay in both glean-api-specs and glean-indexing-api-specs.

Notes on activation

The Custom Metadata path keys (/document/{docId}/custom-metadata/{groupName} and /custom-metadata/schema/{groupName}) are now in source_specs/indexing.yaml upstream (source SHA 503e49e8…), without x-internal: true. So this overlay takes effect immediately when merged.

Test plan

pnpm test — locally verified the overlay file change.

🤖 Generated with Claude Code

…rowing Stacked on #100 — relies on the transformer fix to produce path keys under /rest/api/index, which is what this overlay's targets are written against. - Groups all 5 Custom Metadata endpoints under indexing.customMetadata in generated SDKs and applies clean method names (upsert, delete, getSchema, upsertSchema, deleteSchema). - Introduces a CustomMetadataPropertyDefinition schema that exposes only name, propertyType, and skipIndexing, and re-points CustomMetadataSchema.metadataKeys.items to it. The shared PropertyDefinition schema (used by the Datasource / Custom Properties API) carries fields that are not relevant to Custom Metadata; we cannot strip them from PropertyDefinition itself without affecting the Indexing API surface. - Registers the overlay in both glean-api-specs and glean-indexing-api-specs. The Custom Metadata endpoints themselves are still x-internal: true upstream in scio. This overlay is a no-op until that flag is removed in Phase 2; landing it here ahead of time so the SDK shape is in place when GA flips. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

david-hamilton-glean · 2026-05-13T23:08:03Z

+  # Introduce a Custom Metadata-specific PropertyDefinition schema that exposes only
+  # the fields applicable to Custom Metadata. The shared PropertyDefinition schema
+  # (used by the Datasource / Custom Properties API) carries fields like displayLabel,
+  # displayLabelPlural, uiOptions, hideUiFacet, uiFacetOrder, and group that are not
+  # relevant to Custom Metadata. We cannot strip those fields from PropertyDefinition
+  # itself without affecting the Indexing API surface, so we create a
+  # narrower schema scoped to Custom Metadata and re-point CustomMetadataSchema to it.


This makes sense, but it's unfortunate to need an overlay here.

I presume it's not feasible to change the upstream so that the type that is actually shared is one that has the common properties because the type is currently in use?

It would be quite difficult to manage that since the upstream properties are used in a lot of places both internally and by external API users and having separate structs would require us to make a lot of updates to handle in internal indexing processors.

Additionally - for long term, we would ideally support the UI properties we are applies overlay for here for consistency but it hasn't been prioritized yet.

steve-calvert-glean · 2026-05-13T23:32:55Z

A few things worth flagging before this lands.

1. PR description is stale

The body says:

The Custom Metadata endpoints themselves are still x-internal: true upstream in scio. This overlay is a no-op until that flag is removed in Phase 2.

That's no longer accurate as of today — source_specs/indexing.yaml was updated earlier today (commit 1d56839, source SHA 503e49e8…) and the Custom Metadata path keys (/document/{docId}/custom-metadata/{groupName} and /custom-metadata/schema/{groupName}) are now in the source spec without x-internal: true. So this overlay will take effect immediately when merged on top of #100, not in a follow-up phase. Worth updating the body so the merge audit trail isn't misleading.

2. The CustomMetadataSchema target may not match anything

Grepping the current source spec for any schema named CustomMetadataSchema — it only appears as x-exportParamName: CustomMetadataSchema on a requestBody whose content: {} is empty. There's no actual schema defined under components.schemas.CustomMetadataSchema. So this action:

- target: $["components"]["schemas"]["CustomMetadataSchema"]["properties"]["metadataKeys"]
  update:
    items:
      $ref: "#/components/schemas/CustomMetadataPropertyDefinition"

…has nothing to update. The new CustomMetadataPropertyDefinition schema does get added (the $["components"]["schemas"] action successfully adds a new key), but nothing in the spec ends up $ref-ing it.

Could you confirm whether (a) upstream is going to add a real CustomMetadataSchema definition before the GA target, at which point this narrowing starts working as intended, or (b) the schema is already defined upstream and the local source spec just hasn't picked it up yet? If neither, the narrowing half of this PR is currently dead code.

3. Minor: no overlay test

Not blocking, but tests/post_transform_smoke.test.js would be the natural place to assert that after Speakeasy applies overlays, the five Custom Metadata operations land under x-speakeasy-group: indexing.customMetadata. Otherwise there's no signal if a future overlay reorder or upstream schema rename quietly breaks the SDK grouping.

Asserts that the five Custom Metadata operations exposed by the overlay end up under x-speakeasy-group: indexing.customMetadata with the expected x-speakeasy-name-override values, so a future overlay reorder or upstream schema rename surfaces as a test failure rather than silent SDK drift. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

chinmaya-singal-glean · 2026-05-14T12:50:33Z

@steve-calvert-glean Thanks for the details review

1. PR description
Updated. The body no longer claims the overlay is a no-op; it now spells out that the Custom Metadata path keys are already in source_specs/indexing.yaml and the overlay takes effect on merge.

2. CustomMetadataSchema narrowing
Thanks for pointing it out - I missed removing the x-internal tag from the openapi spec of the schemas - I am sending an update for this so it should exist soon.

3. Overlay smoke test
Added in tests/post_transform_smoke.test.js — asserts that all five Custom Metadata operations carry x-speakeasy-group: indexing.customMetadata and the expected x-speakeasy-name-override values, so an overlay reorder or schema rename can't silently break SDK grouping.

One caveat worth flagging: the transform.yml regen workflow has been failing on main since the Custom Metadata paths landed because Speakeasy lint rejects the empty requestBody.content: {}. That's why the committed overlayed_specs/glean-merged-spec.yaml is stale and doesn't yet contain these paths. The new smoke test will fail PR CI until the upstream content is filled in (which the separate change above should resolve) and the workflow successfully regenerates the merged spec. Will merge this once the spec update is in and the overlay test passes.

Three small fixes that follow from the Custom Metadata paths landing in the source spec with their own path-level server prefix: - overlays/custom-metadata-modifications-overlay.yaml: prettier autofix (single quotes for the $ref string). - tests/post_transform_smoke.test.js: allow `/rest/api/index/` in the base-prefix smoke test so the Custom Metadata path keys (`/rest/api/index/document/...`, `/rest/api/index/custom-metadata/...`) don't trip the prefix check. Without this, the post-transform smoke step in transform.yml fails after Speakeasy successfully regenerates, blocking the commit step and leaving overlayed_specs stale. - tests/source-spec-transformer.test.js: compute the expected basePath per path, mirroring the transformer's path-level-server logic. The previous version assumed every transformed path key was `globalBasePath + originalPath`, which broke once the Custom Metadata paths brought their own `/rest/api/index` server prefix. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

The committed overlayed_specs/glean-merged-spec.yaml is still stale; the assertion will be added back in a follow-up PR once the regen pipeline produces a fresh merged spec on main. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

chinmaya-singal-glean · 2026-05-14T13:14:39Z

I have updated the upstream spec so CustomMetadataSchema is now part of the indexing spec. Reverting the test added here since a different test is failing due to unexpected prefix blocking regeneration of overlay spec.

I will raise a followup with the test instead - also updated the prefix test here to account for custom server

steve-calvert-glean · 2026-05-14T15:23:47Z

If we can get that fix out ASAP that'd be ideal, @chinmaya-singal-glean. I'll make sure the site is fully updated then.

chinmaya-singal-glean · 2026-05-14T15:25:45Z

If we can get that fix out ASAP that'd be ideal, @chinmaya-singal-glean. I'll make sure the site is fully updated then.

@steve-calvert-glean it is already out - the spec on main is also updated in this repo.

chinmaya-singal-glean marked this pull request as ready for review May 7, 2026 16:32

chinmaya-singal-glean requested a review from a team as a code owner May 7, 2026 16:32

david-hamilton-glean approved these changes May 13, 2026

View reviewed changes

Base automatically changed from honor-path-level-servers-in-transformer to main May 13, 2026 23:31

chinmaya-singal-glean and others added 2 commits May 14, 2026 17:11

Merge remote-tracking branch 'origin/main' into custom-metadata-overlay

e93c255

chinmaya-singal-glean and others added 3 commits May 14, 2026 18:28

Merge branch 'main' into custom-metadata-overlay

16bfc6b

chinmaya-singal-glean merged commit 6dadb3d into main May 14, 2026
2 checks passed

chinmaya-singal-glean deleted the custom-metadata-overlay branch May 14, 2026 13:15

chinmaya-singal-glean mentioned this pull request May 14, 2026

Add smoke test for Custom Metadata SDK grouping #105

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Custom Metadata overlay for SDK groups and PropertyDefinition narrowing#101

Add Custom Metadata overlay for SDK groups and PropertyDefinition narrowing#101
chinmaya-singal-glean merged 6 commits into
mainfrom
custom-metadata-overlay

chinmaya-singal-glean commented May 7, 2026 •

edited

Loading

Uh oh!

david-hamilton-glean May 13, 2026

Uh oh!

chinmaya-singal-glean May 14, 2026

Uh oh!

steve-calvert-glean commented May 13, 2026

Uh oh!

chinmaya-singal-glean commented May 14, 2026

Uh oh!

chinmaya-singal-glean commented May 14, 2026

Uh oh!

Uh oh!

steve-calvert-glean commented May 14, 2026

Uh oh!

chinmaya-singal-glean commented May 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

chinmaya-singal-glean commented May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Notes on activation

Test plan

Uh oh!

david-hamilton-glean May 13, 2026

Choose a reason for hiding this comment

Uh oh!

chinmaya-singal-glean May 14, 2026

Choose a reason for hiding this comment

Uh oh!

steve-calvert-glean commented May 13, 2026

Uh oh!

chinmaya-singal-glean commented May 14, 2026

Uh oh!

chinmaya-singal-glean commented May 14, 2026

Uh oh!

Uh oh!

steve-calvert-glean commented May 14, 2026

Uh oh!

chinmaya-singal-glean commented May 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

chinmaya-singal-glean commented May 7, 2026 •

edited

Loading